Optimization of Reference-less Evaluation Metric of Grammatical Error Correction for Manual Evaluations
نویسندگان
چکیده
منابع مشابه
There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metr...
متن کاملReference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems
In grammatical error correction (GEC), automatically evaluating system outputs requires gold-standard references, which must be created manually and thus tend to be both expensive and limited in coverage. To address this problem, a referenceless approach has recently emerged; however, previous reference-less metrics that only consider the criterion of grammaticality, have not worked as well as ...
متن کاملBetter Evaluation for Grammatical Error Correction
We present a novel method for evaluating grammatical error correction. The core of our method, which we call MaxMatch (M), is an algorithm for efficiently computing the sequence of phrase-level edits between a source sentence and a system hypothesis that achieves the highest overlap with the goldstandard annotation. This optimal edit sequence is subsequently scored using F1 measure. We test our...
متن کاملHuman Evaluation of Grammatical Error Correction Systems
The paper presents the results of the first large-scale human evaluation of automatic grammatical error correction (GEC) systems. Twelve participating systems and the unchanged input of the CoNLL-2014 shared task have been reassessed in a WMT-inspired human evaluation procedure. Methods introduced for the Workshop of Machine Translation evaluation campaigns have been adapted to GEC and extended...
متن کاملGrammatical Error Correction with Alternating Structure Optimization
We present a novel approach to grammatical error correction based on Alternating Structure Optimization. As part of our work, we introduce the NUS Corpus of Learner English (NUCLE), a fully annotated one million words corpus of learner English available for research purposes. We conduct an extensive evaluation for article and preposition errors using various feature sets. Our experiments show t...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Journal of Natural Language Processing
سال: 2021
ISSN: 1340-7619,2185-8314
DOI: 10.5715/jnlp.28.404